Closes apache#5439 from pitrou/ARROW-3777-slow-fs and squashes the following commits:

2ca64c5 <Antoine Pitrou> Try to fix Windows failure
b02a8c5 <Antoine Pitrou> ARROW-3777:  Add Slow input streams and slow filesystem

Authored-by: Antoine Pitrou <antoine@python.org>
Signed-off-by: Antoine Pitrou <antoine@python.org>

…cal plan

This PR adds the binary expression to the new physical execution plan, with support for comparison operators (`<`, `<=`, `>`, `>=`, `==`, `!=`) and boolean operators `AND` and `OR`.

Other binary expressions, such as math expressions will be added in a future PR.

Closes apache#5478 from andygrove/ARROW-6669 and squashes the following commits:

83bfa77 <Andy Grove> formatting
af8d298 <Andy Grove> address PR feedback
9ad3b7f <Andy Grove> formatting
bb82a24 <Andy Grove> use expect() instead of unwrap() when downcasting arrays
9b94cc8 <Andy Grove> Implement binary expression with support for comparison and boolean operators

Authored-by: Andy Grove <andygrove73@gmail.com>
Signed-off-by: Paddy Horan <paddyhoran@hotmail.com>

… to Parquet

https://issues.apache.org/jira/browse/ARROW-6187

Closes apache#5436 from jorisvandenbossche/ARROW-6187-parquet-extension-type and squashes the following commits:

e56164b <Joris Van den Bossche> expose constants in extension_type.h
61d245e <Joris Van den Bossche> clean-up chunked array creation
6b2f190 <Joris Van den Bossche> recreate extension type on read
bdda0f7 <Joris Van den Bossche> test that extension metadata is already saved
abf2a2f <Joris Van den Bossche> add python test
fb4b810 <Joris Van den Bossche> ARROW-6187:  Fallback to storage type when writing ExtensionType to Parquet

Authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
Signed-off-by: Antoine Pitrou <antoine@python.org>

Also adds and improves other C++ doc items.

Closes apache#5487 from pitrou/ARROW-6629-cpp-fs-docs and squashes the following commits:

d47a008 <Antoine Pitrou> Update docs/source/cpp/io.rst
895f04a <Antoine Pitrou> Try to fix Sphinx error on Travis
c40e0e2 <Antoine Pitrou> ARROW-6629:   Add filesystem docs

Authored-by: Antoine Pitrou <antoine@python.org>
Signed-off-by: Neal Richardson <neal.p.richardson@gmail.com>

…atch

See the readme and the new tests for example output.

This patch also fixes a validation bug in `dictionary()`, aliases that to `DictionaryType$create`, and adds default arguments.

Closes apache#5492 from nealrichardson/print-methods and squashes the following commits:

c092d30 <Neal Richardson> Merge branch 'print-methods' of github.com:nealrichardson/arrow into print-methods
02afb89 <Neal Richardson> Prettier printing of dictionary type's ordered attribute
5750100 <Neal Richardson> indices in the docs too
6be0328 <Neal Richardson> indices
2d4e744 <Neal Richardson> Add/improve print methods for Array, ChunkedArray, Table, RecordBatch

Authored-by: Neal Richardson <neal.p.richardson@gmail.com>
Signed-off-by: Neal Richardson <neal.p.richardson@gmail.com>

https://issues.apache.org/jira/browse/ARROW-6674

Closes apache#5489 from jorisvandenbossche/ARROW-6674-test-warnings and squashes the following commits:

2a2bb14 <Joris Van den Bossche> ARROW-6674:  Fix or ignore the test warnings

Authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
Signed-off-by: Wes McKinney <wesm+git@apache.org>

…of StructArray

https://issues.apache.org/jira/browse/ARROW-6158

Closes apache#5488 from jorisvandenbossche/ARROW-6158-struct-array-validation and squashes the following commits:

7573781 <Joris Van den Bossche> ARROW-6158:  Validate child array types with type fields of StructArray

Authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
Signed-off-by: Wes McKinney <wesm+git@apache.org>

…t be base64-encoded to be UTF-8 compliant

I have added a simple base64 implementation (Zlib license) to arrow/vendored from

https://github.com/ReneNyffenegger/cpp-base64

Closes apache#5493 from wesm/ARROW-6678 and squashes the following commits:

c058e86 <Wes McKinney> Simplify, add MSVC exports
06f75cd <Wes McKinney> Fix Python unit test that needs to base64-decode now
eabb121 <Wes McKinney> Fix LICENSE.txt, add iwyu export
b3a584a <Wes McKinney> Add vendored base64 C++ implementation and ensure that Thrift KeyValue in Parquet metadata is UTF-8

Authored-by: Wes McKinney <wesm+git@apache.org>
Signed-off-by: Micah Kornfield <emkornfield@gmail.com>

…n" operator

This PR implements the physical execution plan for the selection operator (the WHERE clause in a SQL query).

In order to have working tests, I also had to implement some subset of expressions (column reference, literal value, comparison expressions, and CAST). However, the goal of this PR is not to add complete support for all expressions but to implement the Selection operator. I will create separate JIRA/PRs for adding support for other expressions and data types in the physical query plan.

Closes apache#5320 from andygrove/ARROW-6089 and squashes the following commits:

6cad327 <Andy Grove> Implement selection operator

Authored-by: Andy Grove <andygrove73@gmail.com>
Signed-off-by: Andy Grove <andygrove73@gmail.com>

…quet

_build_nested_path has a reference cycle because the closured function refers to the parent cell which also refers to the closured function again. Address this by clearing the reference to the function from the parent cell before returning.

open_dataset_file is partialed with self inside the ParquetFile class. Prevent this by using a weakref instead.

Closes apache#5476 from AaronOpfer/master and squashes the following commits:

f4909e0 <Wes McKinney> Fix flakes
883ab86 <Aaron Opfer> ARROW-6667:  remove cyclical object references in pyarrow.parquet

Lead-authored-by: Aaron Opfer <aaron.opfer@chicagotrading.com>
Co-authored-by: Wes McKinney <wesm+git@apache.org>
Signed-off-by: Antoine Pitrou <antoine@python.org>

- Adds StatusDetail to the docs
- Fixes the Doxygen config to work with `ARROW_FLIGHT_EXPORT`
- Adds error codes to the format docs (though they're not part of the formal format)
- Touches up some docstrings and adds missing classes to the API docs
- Add a basic description of how to set up a Flight server/client in C++

Closes apache#5491 from lihalite/flight-docs and squashes the following commits:

2948983 <David Li> ARROW-6677:  Document Flight in C++

Authored-by: David Li <li.davidm96@gmail.com>
Signed-off-by: Antoine Pitrou <antoine@python.org>

…taframe

- add scanReverse() to dataFrame and filteredDataframe
- add tests for scanReverse()

Closes apache#5480 from mmaclach/master and squashes the following commits:

01faae8 <mmaclach> JS: scanReverse

Authored-by: mmaclach <mmaclachlan@ccri.com>
Signed-off-by: Krisztián Szűcs <szucs.krisztian@gmail.com>

I fixed this in R, though I wonder if that's not enough. At a minimum, the C++ docs should note this requirement (@pitrou maybe you know the best place in the docs to add this?), and I still think it would be nice if all of this path normalization were handled in C++ (cf. https://issues.apache.org/jira/browse/ARROW-6324).

Closes apache#5445 from nealrichardson/win-fs-fix and squashes the following commits:

515d710 <Neal Richardson> Rename functions
1775a6d <Neal Richardson> Fix two other absolute paths I missed
0dce0d7 <Neal Richardson> Munge paths directly on windows
f03815e <Neal Richardson> Normalize paths for filesystem API on Windows

Authored-by: Neal Richardson <neal.p.richardson@gmail.com>
Signed-off-by: Neal Richardson <neal.p.richardson@gmail.com>

…ata sources

I discovered this last minute while running manual tests. I have been able to run parallel queries against parquet files using this branch as a dependency.

Closes apache#5494 from andygrove/ARROW-6086 and squashes the following commits:

77bee15 <Andy Grove> Replace unwrap with Result combinator
cd11b97 <Andy Grove> don't panic
c751753 <Andy Grove> Add support for partitioned parquet data sources
25eaf45 <Andy Grove> Move build_file_list into common module

Authored-by: Andy Grove <andygrove73@gmail.com>
Signed-off-by: Andy Grove <andygrove73@gmail.com>

cf. https://github.com/jeroen/autobrew/blob/gh-pages/LICENCE.txt

Closes apache#5501 from nealrichardson/autobrew-license and squashes the following commits:

3e790f5 <Neal Richardson> MIT license for autobrew

Authored-by: Neal Richardson <neal.p.richardson@gmail.com>
Signed-off-by: Wes McKinney <wesm+git@apache.org>

Closes apache#5500 from pitrou/ARROW-6630-file-format-docs and squashes the following commits:

aa5c57d <Antoine Pitrou> ARROW-6630:  Document C++ file formats

Authored-by: Antoine Pitrou <antoine@python.org>
Signed-off-by: Antoine Pitrou <antoine@python.org>

Related to [ARROW-6472](https://issues.apache.org/jira/browse/ARROW-6472).

If we use visitor API this way:
>RangeEqualsVisitor visitor = new RangeEqualsVisitor(vector1, vector2);
vector3.accept(visitor, range)

if vector1/vector2 are say, StructVector}}s and vector3 is an {{IntVector - things can go bad. we'll use the compareBaseFixedWidthVectors() and do wrong type-casts for vector1/vector2.

Discussions see:
apache#5195 (comment)
https://issues.apache.org/jira/browse/ARROW-6472

Closes apache#5483 from tianchen92/ARROW-6472 and squashes the following commits:

3d3d295 <tianchen> add test
12e4aa2 <tianchen> ARROW-6472:  ValueVector#accept may has potential cast exception

Authored-by: tianchen <niki.lj@alibaba-inc.com>
Signed-off-by: Pindikura Ravindra <ravindra@dremio.com>

Initial support for array reader. List and map support will come later.

Closes apache#5378 from liurenjie1024/arrow-4218 and squashes the following commits:

433abab <Renjie Liu> Remove unwraps with result
6407dee <Renjie Liu> Fix format
4dd9a01 <Renjie Liu> Initial support for array reader
215f73b <Renjie Liu> struct array reader
2e898ff <Renjie Liu> test done for primitive array reader

Authored-by: Renjie Liu <liurenjie2008@gmail.com>
Signed-off-by: Andy Grove <andygrove73@gmail.com>

Closes apache#5503 from andygrove/ARROW-6687 and squashes the following commits:

c0ded56 <Andy Grove> Bug fix in DataFusion Parquet reader

Authored-by: Andy Grove <andygrove73@gmail.com>
Signed-off-by: Andy Grove <andygrove73@gmail.com>

Closes apache#5499 from alippai/patch-1 and squashes the following commits:

d2fd03e <Ádám Lippai> Update README.md

Authored-by: Ádám Lippai <adam@rigo.sk>
Signed-off-by: Andy Grove <andygrove73@gmail.com>

…able

This is needed to use correct download URL for RC.

Closes apache#5506 from kou/packaging-linux-restore-arrow-version and squashes the following commits:

7face8a <Sutou Kouhei>  Restore ARROW_VERSION environment variable

Authored-by: Sutou Kouhei <kou@clear-code.com>
Signed-off-by: Micah Kornfield <emkornfield@gmail.com>

Related to [ARROW-6709](https://issues.apache.org/jira/browse/ARROW-6709).
Currently, in several consumers, currentIndex only increments when ResultSet was not null.
However, if ResultSet contains null values, the Arrow vector valueCount is not correct.

Closes apache#5511 from tianchen92/ARROW-6709 and squashes the following commits:

b1e9d5a <tianchen> ARROW-6709:  Jdbc adapter currentIndex should increment when value is null

Authored-by: tianchen <niki.lj@alibaba-inc.com>
Signed-off-by: Micah Kornfield <emkornfield@gmail.com>

Construct tree structure from std::vector<fs::FileStats>, following the path directory hierarchy.

Closes apache#5430 from fsaintjacques/ARROW-6606-path-tree and squashes the following commits:

43d19fa <François Saint-Jacques> Address comments
60b5945 <François Saint-Jacques> Simplify implementation
109ea85 <François Saint-Jacques> ARROW-6606:  Add PathTree tree structure

Authored-by: François Saint-Jacques <fsaintjacques@gmail.com>
Signed-off-by: Benjamin Kietzman <bengilgit@gmail.com>

…lity

https://issues.apache.org/jira/browse/ARROW-6683

@wesm is this more or less what you were thinking?

Closes apache#5498 from jorisvandenbossche/ARROW-6683-fastparquet-cross-testing and squashes the following commits:

4c1e3aa <Joris Van den Bossche> add comment
1d35f3b <Joris Van den Bossche> add fastparquet mark
c5d6161 <Joris Van den Bossche> ARROW-6683:  Test for fastparquet <-> pyarrow cross-compatibility

Authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
Signed-off-by: Krisztián Szűcs <szucs.krisztian@gmail.com>

…ntegrition tests

Arrow Java Writer now requires an IpcOption for some APIs, this patch fixes the compilation to run Spark Integration tests.

Closes apache#5465 from BryanCutler/spark-integration-patch-ARROW-6429 and squashes the following commits:

918ab91 <Bryan Cutler> Remove redundant message
655937d <Bryan Cutler> Changes to Spark integration test config
dd2483f <Bryan Cutler> Add patch to rat excludes
48b2eac <Bryan Cutler> Adding patch to fix Spark compilation with IpcOption

Authored-by: Bryan Cutler <cutlerb@gmail.com>
Signed-off-by: Bryan Cutler <cutlerb@gmail.com>

Rust 1.40.0-nightly just got released and caused builds to start failing

Closes apache#5519 from andygrove/ARROW-6716 and squashes the following commits:

5301d7b <Andy Grove> trigger rebuild
b651d7b <Andy Grove> Use 1.40.0-nightly-2019-09-25

Authored-by: Andy Grove <andygrove73@gmail.com>
Signed-off-by: Paddy Horan <paddyhoran@hotmail.com>

…row specific)

This adds parameters to `write_parquet()` to control compression, whether to use dictionary, etc ... on top of the C++ classes `parquet::WriterProperties` and `parquet::ArrowWriterProperties` e.g.

```r
write_parquet(tab, file, compression = "gzip", compression_level = 7)
```

Closes apache#5451 from romainfrancois/ARROW-6532/write_parquet_compression and squashes the following commits:

413dd41 <Romain Francois> test make_valid_version()
50555f8 <Romain Francois> rename arguments to `x` and `sink`
9aff79b <Romain Francois> implement ==.Object that calls $Equals instead of implementing for each class.
ecd9218 <Romain Francois> rework documentation for write_parquet()
56dac33 <Romain Francois> Move read_parquet() and write_parquet() to top of the file
45ec63b <Romain François> Update r/R/parquet.R
66c51fd <Romain Francois> added all.equal.Object() that uses ==
c5549de <Romain Francois> Test ==.Table
5ade52d <Romain Francois> wrong length for use_dictionary and write_statistics
00cc214 <Romain Francois> abstract various ParquetWriterPropertiesBuilder$set_*() methods
1fdcc0b <Romain Francois> suggestsions from @nealrichardson
9bee8de <Romain Francois> define and use internal make_valid_version() function
004cf90 <Romain Francois> M%ake compression_from_name() vectorized
86d9ff4 <Romain Francois> Remove the _ from builder classes
6c4f003 <Romain Francois> add test helper so that we actually can test parquet roundtrip
d318a66 <Romain Francois> ==.Table
7f1c184 <Romain Francois> align arguments following tidyverse style guide
72caaab <Romain Francois> using assert_that()
738ea6e <Romain Francois> Remove $default() methods and use $create() wityh default arguments instead.
1166264 <Romain Francois> using make_valid_time_unit()
4055f67 <Romain Francois> More flexible arguments use_dictionary= and write_statistics=
2f2ae00 <Romain Francois> More flexible compression= and compression_level=
1e3b5b6 <Romain Francois> document()
2dd2cb9 <Romain Francois> + compression_level= in write_parquet()
b8337e1 <Romain Francois> lint
fa8990b <Romain Francois> Expose options from ParquetWriterProperties and ParquetArrowWriterProperties to write_parquet()
09ea0ad <Romain Francois> + ParquetWriterProperties$create() and associated ParquetWriterProperties_Builder class skeleton
1b84ad4 <Romain Francois> Exposing classes parquet::arrow::ArrowWriterProperties and parquet::arrow::WriterProperties to R side
0e09ac8 <Romain Francois> lint
aa34095 <Romain Francois> passing down the right stream
9ed32b6 <Romain Francois> Make write_parquet() generic, internal impl using streams rather than file path for more flexibility

Lead-authored-by: Romain Francois <romain@rstudio.com>
Co-authored-by: Romain François <romain@purrple.cat>
Signed-off-by: Neal Richardson <neal.p.richardson@gmail.com>

Closes apache#5514 from nealrichardson/fix-lint and squashes the following commits:

ea8b8e7 <Neal Richardson> Note that clang-format-7 is required
f83f0e1 <Neal Richardson> Incorporate @kou's suggestion
6bcc063 <Neal Richardson> Note how to use lint.sh and have it look for clang-format rather than hard-code its location

Authored-by: Neal Richardson <neal.p.richardson@gmail.com>
Signed-off-by: Neal Richardson <neal.p.richardson@gmail.com>

I noticed this now-defunct call to `table()` while reviewing another PR. We clearly weren't testing this case because if you were to pass a data.frame in, you'd get a segfault. This patch adds tests and fixes the issue.

Closes apache#5518 from nealrichardson/record-batch-writer-fix and squashes the following commits:

afad9fe <Neal Richardson> Fix untested RecordBatchWriter case

Authored-by: Neal Richardson <neal.p.richardson@gmail.com>
Signed-off-by: Neal Richardson <neal.p.richardson@gmail.com>